Tensor Displays

ABSTRACT

In exemplary implementations of this invention, an automultiscopic display device includes (1) one or more spatially addressable, light attenuating layers, and (2) a controller which is configured to perform calculations to control the device. In these calculations, tensors provide sparse, memory-efficient representations of a light field. The calculations include using weighted nonnegative tensor factorization (NTF) to solve an optimization problem. The NTF calculations can be sufficiently efficient to achieve interactive refresh rates. Either a directional backlight or a uniform backlight may be used. For example, the device may have (1) a high resolution LCD in front, and (2) a low resolution directional backlight. Or, for example, the device may have a uniform backlight and three or more LCD panels. In these examples, all of the LCDs and the directional backlight (if applicable) may be time-multiplexed.

RELATED APPLICATIONS

This application is a non-provisional of, and claims the benefit of thefiling date of, U.S. Provisional Application Ser. No. 61/590,507, filedJan. 25, 2012, the entire disclosure of which is herein incorporated byreference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with U.S. government support under: (i) grantIIS-1116452, awarded by the National Science Foundation; and (ii) awards10-DARPA-1102, N66001-10-1-4041, and P.O. 10320917, awarded by theDefense Advanced Research Projects Agency. The government has certainrights in this invention.

FIELD OF THE TECHNOLOGY

The present invention relates generally to 3D displays.

SUMMARY

In exemplary implementations of this invention, a display deviceproduces a 3D image. The device comprises a stack ofspatially-addressable light attenuating layers.

A controller performs calculations in order to control the device. Inthese calculations, tensors provide sparse, memory-efficientrepresentations of a light field. The calculations include usingweighted nonnegative tensor factorization (NTF) to solve an optimizationproblem. The tensor calculations may employ multiplicative update rules,and may be sufficiently efficient to achieve interactive refresh rates.

In exemplary implementations of this invention, a tensor display has Nspatially addressable light attenuating layers and displays a temporalsequence of M frames. If the tensor display has a uniform backlight, thelight field that it emits may be represented by N^(th)-order, rank-Mtensor. If the tensor display has a directional backlight, the lightfield that it emits may be represented by a rank-M tensor with an orderequal to N+1 (where N does not count any spatially addressable lightattenuating layers in the directional backlight itself). The controllerperforms calculations that involve such a tensor.

In exemplary implementations, the display device is automultiscopic: aviewer can see the 3D effect without wearing glasses. The 3D displayappears different from different viewing angles, and thus exhibits bothbinocular disparity and motion parallax, which are cues for depthperception. Thus, the display device creates the illusion of looking atan actual 3D scene.

In exemplary implementations, the display device includes (1) at leastone spatially addressable light attenuating layer, and (2) a backlight.For example, each layer may comprise a liquid crystal display (LCD)layer. Light from the backlight is transmitted through the layer(s). Thelayer(s) are time-multiplexed: i.e., configured to produce atime-multiplexed image, in which a sequence of frames is shown at a rateequal to or faster than the flicker fusion frequency. This causes ahuman viewer to perceive a time average of the frames.

In exemplary implementations of this invention, the controller comprisesone or more processors. Using weighted NTF, the controller calculates anoptimal set of attenuations induced in light at respective pixels in thelayers in the stack. If the backlight is directional, the optimal setmay also include angles of light rays emitted by respective pixels ofthe backlight. The controller outputs control signals that cause thedisplay device to produce this optimal set of attenuations (and lightray angles, if applicable) on a per pixel basis. Thus, the controllercauses the device to output a light field that optimally approximatesthe target light field. If a human looks at the outputted light fieldwithout special glasses, the human perceives a 3D image.

In exemplary implementations of this invention, the directionalbacklight is effectively a low resolution light field display. Thisdirectional backlight can control the angle of light that it emits, andcan be used with an arbitrary number of light attenuator layers (e.g.,one or more such layers).

Here are two examples of a tensor display with a directional backlight.

In the first example, a purely angular backlight illuminates a singlespatially addressable, light attenuating layer. The light attenuatorlayer comprises an LCD layer. The purely angular backlight can vary theangle of light that it emits, but does not have spatially addressablelight attenuating pixels. Both the light attenuator layer and the purelyangular backlight are time-multiplexed.

In the second example, the tensor display includes two spatiallyaddressable light attenuator layers: (1) a high spatial resolution LCDin front, and (2) a secondary LCD layer in back. The secondary LCD layeris part of the directional backlight. Again, both the front LCD and thebacklight are time-multiplexed.

In this second example, the directional backlight may comprise, fromback to front: (1) an array of cold-cathode fluorescent lamps (CCFLs);(2) one or more lenses; and (3) the secondary LCD layer, positioned at adistance from the lens(es) equal to the focal length of the lens(es). Bycontrolling which of the CCFLs behind the lens(es) are lit, the angle oflight transmitted through respective pixels in the secondary LCD can becontrolled.

Alternately, other types of light emitters may be used in thedirectional backlight. For example, the CCFLs may be replaced with otherlight-emitting devices, including light-emitting diodes (LEDs),including organic light-emitting diodes (OLEDs). Alternately, the stackof light attenuating layers may comprise two or more LCD layers, inaddition to any LCD layer that is part of the directional backlight.

In some implementations, the display device is illuminated by a uniformbacklight. For example, the stack may comprise two, three or morelayers, where each layer is a spatially addressable light attenuator andnone of the layers are part of the uniform backlight.

As used herein, a “tensor display” is an automultiscopic display devicewherein: (1) the device includes one or more spatially addressable,light attenuating layers; (2) the device includes a controller, which isconfigured to perform calculations to control the device; and (3) thecalculations involve using weighted NTF.

In exemplary implementations of this invention, time-multiplexed tensordisplays have many practical advantages. For example, such tensordisplays can be brighter than existing automultiscopic displays.Further, such tensor displays may have wider fields of views, greaterdepths of field and thinner form factors than existing automultiscopicdisplays. For these reasons, such tensor displays are well suited forproducing 3D displays in mobile devices, such as tablets, smartphonesand cell phones. More generally, tensor displays may be used in any flatscreen display device, including in a monitor for (i) a personalcomputer (PC), (ii) a laptop computer, or (iii) home theater.

Tensor-based calculations can be used in a wide variety ofarchitectures. Among other things, these calculations can be used tocontrol an arbitrary number of light attenuating layers (e.g., two,three, four or more layers) and, if the backlight is directional, tocontrol angle of light emitted by the backlight.

The description of the present invention in the Summary and Abstractsections hereof is just a summary. It is intended only to give a generalintroduction to some illustrative implementations of this invention. Itdoes not describe all of the details of this invention. This inventionmay be implemented in many other ways.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart for a tensor display.

FIG. 2 is a diagram of a tensor display illuminated by a directionalbacklight.

FIG. 3 is a diagram of three layer tensor display, illuminated by auniform backlight.

FIG. 4 shows tensor display coordinates.

FIG. 5 is an overview of tensor displays. The overview shows (i) anoriginal (input) light field for a scene, (ii) visualizations of thelight field, for different tensor display architectures, and (iii)reconstructed views of the scene produced by different tensor displayarchitectures.

FIG. 6 shows reconstructed views of a scene produced by tensor displays(left), and tensor display decompositions of the scene (right).

FIG. 7 shows an original light field for a scene (left), images of thescene produced by integral imaging (middle), and images of the sceneproduced by a tensor display (right).

FIG. 8 shows an original light field of a scene (top row) and tensordisplay decompositions of the scene (other rows).

FIG. 9 shows NTF-based multilayer decompositions of a scene.

FIG. 10 shows multilayer decompositions of a scene, using a directionalbacklight.

The above Figures illustrate some illustrative implementations of thisinvention, or provide information that relates to those implementations.However, this invention may be implemented in many other ways. The aboveFigures do not show all of the details of this invention.

DETAILED DESCRIPTION

In exemplary implementations, this invention comprises a tensor display,i.e., an automultiscopic display device wherein: (1) the device includesone or more spatially addressable, light attenuating layers; (2) thedevice includes a controller, which is configured to performcalculations to control the device; and (3) the calculations involveusing weighted NTF. Tensor displays can be illuminated by either uniformor directional backlighting (e.g. a low-resolution light field emitter).

In exemplary implementations of this invention, an N-layer, M-frametensor display is illuminated by a uniform backlight, and the lightfield that it emits can be represented by an N^(th)-order, rank-Mtensor. The light field tensor is decomposed as a sum of M rank-1tensors, each corresponding to the outer product of N masks representingthe transmittance of each layer for each frame. (If the backlight isdirectional, then the tensor display emits a light field that can berepresented by N+1-order, rank-M tensor, where N does not count anyspatially addressable light attenuating layers in the directionalbacklight itself). Using this representation, a unified optimizationframework based on nonnegative tensor factorization (NTF) may be appliedto a wide variety of tensor display architectures. For example, this NTFoptimization framework may be employed for tensor display architecturescomprising: (1) multiple, time-multiplexed layers with directionalbacklighting; (2) multiple, time-multiplexed layers with uniformbacklighting, (3) static, multilayer displays with three or more layersand uniform backlighting, (4) two-layer, temporally-multiplexeddisplays, or (5) a full resolution 2D mode (in which all but one of thelayers is rendered transparent).

Advantageously, this NTF optimization framework allows joint multilayer,multiframe light field decompositions. Such decompositions significantlyreduce artifacts observed with prior multilayer-only and multiframe-onlydecompositions.

In a prototype of this invention, a tensor display includes modified LCDpanels and a custom integral imaging backlight. In this prototype, anefficient, GPU-based NTF implementation enables interactiveapplications.

In exemplary implementations, the one or more layers act as acompressive display. Imagery sent to the display is compressed. But itis compressed in a way that allows the viewer to at least partiallydecompress it. In a stack with N spatially addressable, lightattenuating layers, any light field sent to the display is reduced to Nimages—one for each of the layers. As these images are presented at ahigh frame rate, the viewer's eyes integrates the images into ahigh-rank approximation of the desired light field, thereby at leastpartially decompressing the imagery The higher the rank approximationthat is achieved, the less lossy the compression.

In exemplary implementations of this invention, directional backlighting(used with multiple, time-multiplexed light attenuating layers) achievesa wide field of view, while reducing the need for additional layers andframes, yielding a thin, power-efficient, high-resolution light fielddisplay well suited for mobile and home theater applications. Forexample, in a prototype, a low-resolution lenslet-based directionalbacklight is used with a high-resolution LCD. In this prototype, atarget light field is decomposed into a low-rank tensor approximation,increasing brightness and allowing more views to be generated thanavailable frames. The NTF optimization framework allows arbitrarycombinations of directional backlights and multiple light-attenuatingdisplay layers.

In exemplary implementations of this invention, multiplicative lightattenuation to allow synthesized 3D objects to extend outside theenclosure. Furthermore, tensor displays support specularities,occlusions, and global illumination effects, without requiring movingparts.

In exemplary implementations, tensor displays enable trade-offs betweenimage fidelity, resolution, brightness, and display complexity. Thesetensor displays employ compressive display modes, wherein low-ranktensor approximation efficiently exploits correlations betweenneighboring views to synthesize an emitted light field with an apparentnumber of views exceeding the number of frames. In contrast, priordirect display modes assign a single view to each frame, limitingresolution and brightness.

Tensor displays can provide greater depths of field, wider fields ofview, and thinner form factors, compared to prior automultiscopicdisplays.

FIG. 1 is a high-level flow chart for a tensor display. A controlleremploys a linear optimization algorithm 101 to determine time changingattenuation mask patterns 109 for respective layers in a stack oftemporally multiplexed, light attenuating layers. The controller outputscontrol signals to cause the layers to display these masks. Adirectional or uniform backlight illuminates the masks 111. Light fromthe backlight is transmitted through the layers in the stack, andemerges from the tensor display as a light field that can be perceivedas a 3D image by a human viewer, without the viewer wearing any specialglasses or optical apparatus 113. The target light field may compriseeither (i) captured multi-view or light field data 105 or (ii) syntheticmulti-view or light field data 103. In the latter case, a 3D model 102may be used when generating the synthetic data 103. The target lightfield (103 or 105) is an input to the optimization algorithm 101. Theoptimization algorithm 101 may employ simplifying assumptions, and maybe customized for (or may take into account) hardware designspecification 107.

FIG. 2 is a diagram of a cross-section of a tensor display illuminatedby a directional backlight 203. The tensor display includes two LCDlayers 201, 211. One of these LCD layers 201 is part of the directionalbacklight 203. The directional backlight 203 comprises: (i) a lightsource 221 (e.g. a CCFL array or LED array); (ii) polarization filters(e.g., polarizing films) 219, 217, (iii) a time-multiplexed LCD layer213, and (iv) a sheet of lenslets 215. The directional backlight 203illuminates a high resolution, time-multiplexed LCD layer 201.Polarization filters (e.g., polarizing filters) 205, 209 are positionedadjacent to LCD layer 201, one on each side of LCD layer 201.Polarization filters 205 and 207 are crossed (relative to each other),and polarization filters 217 and 219 are crossed (relative to eachother). A controller 225 controls the masks displayed by LCD layers 201,211. In addition, controller 225 controls the angle of light produced byrespective pixels in the directional backlight 203. Alternately,directional backlight 203 may be a purely angular backlight.

FIG. 3 is a diagram of a cross section of a three layer tensor display,illuminated by a uniform backlight. In this example, none of the layersis part of the uniform backlight. As shown in FIG. 3, a uniformbacklight 301 illuminates a stack of three LCDs 303, 305, 307.Polarization filters (e.g., polarization films) 315, 317, 319, 321 arearranged so that the polarization angles of adjacent filters areorthogonal to each other.

LCD layers 201, 211, 303, 305, 307 are configured to be able totemporally vary the light attenuation on a per pixel basis. For example,for each frame in a temporal sequence of frames, the controller (225 or325) can control, for each respective each pixel (e.g., 209, 213, 309,311, 313) in the LCD layers, whether light is more or less attenuated asit is transmitted through the pixel.

In the examples shown in FIGS. 2 and 3, the controller (e.g., 225, 325)comprises one or more processors. These processors perform computations,including rendering and numerical calculations of optimizedapproximations (e.g., using NT and multiplicative update). For example,the processors may compute time-multiplexed optimal masks, and mayoutput command signals for the respective LCDs to display these masks.The processors are connected to the stack of LCDs by one or moreconnections, which connections may be wired or wireless. Depending onthe particular implementation, the location of the one or moreprocessors may vary. For example, the one or more processors may all beremote from the stack of LCDs. Or, for example, at least some of the oneor more processors may be housed with, or adjacent to, the stack ofLCDs.

FIG. 4 is a diagram that illustrates tensor display coordinates. Tensordisplays comprise a stack of N light-attenuating layers illuminated byeither a uniform backlight or a directional backlight. In the exampleshown in FIG. 4, there are three spatially addressable,light-attenuating layers 403, 405, 407. For full generality, assume thatdisplay layers support synchronized, high-speed temporal modulation,such that an observer perceives the time average of an M-framemultilayer mask sequence.

First, consider 2D light fields and 1D layers (an extension to 4D lightfields and 2D layers is covered later). FIG. 4 shows a relativetwo-plane light field parameterization l(x, v), where v denotes thepoint of intersection of the ray (x, v) with a plane located a distanced_(r) from the x-axis, expressed relative to x. The bottom left side ofFIG. 4 represents one example of a directional backlight (implemented bycovering a uniform backlight 401 with a lenslet array 415 and an LCDlayer 403). The bottom right side of FIG. 4 represents a uniformbacklight illuminating a stack of LCD layers 403, 405, 407.

Static Multilayer Displays: Consider a fixed stack of Nlight-attenuating layers (i.e., one that does not support temporalvariation of the mask patterns). When illuminated by a uniform backlightwith unit radiance, the emitted light field {tilde over (l)}(x, v) isgiven by the following expression:

$\begin{matrix}{{\overset{\sim}{l}\left( {x,{v;N}} \right)} = {\prod\limits_{n = 1}^{N}\; {f^{(n)}\left( {x + {\left( {d_{n}/d_{r}} \right)v}} \right)}}} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$

where ƒ^((n))(ξ_(n))ε[0,1] is the transmittance at the point ξ_(n) oflayer n, separated a distance d_(n) from the x-axis.

Consider a three-layer configuration, with the transmittances for therear, middle, and front layers given by ƒ(ξ₁), g(ξ₂), and h(ξ₃),respectively. Equation 1 gives the following expression for the emittedlight field.

{tilde over (l)}(x,v)=ƒ(ξ₁)g(ξ₂)h(ξ₃), for ξ_(n) =x+(d _(n) /d_(r))v  (Eq. 2)

The emitted light field {tilde over (l)}(x,v) can be represented as therestriction of the function

{tilde over (t)}(ξ₁,ξ₂,ξ₃)=ƒ(ξ₁)g(ξ₂)h(ξ₃)  (Eq. 3)

defined in the three-dimensional Euclidean space

³ spanned by {ξ₁, ξ₂, ξ₃}, to the two-dimensional subspace defined bythe equation αξ₁+βξ₂+γξ₃=0, with

α=d ₃ −d ₂ , β=d ₁ −d ₃ , γ=d ₂ −d ₁  (Eq. 4)

Thus, elements of the emitted light field {tilde over (l)}(x,v) arerestricted to the plane corresponding to Equation 4.

For the general case with N>3 layers, the emitted light field {tildeover (l)}(x,v) can also be represented as the restriction of thefunction

${{\overset{\sim}{t}\left( {\xi_{1},\xi_{2},\ldots \mspace{14mu},\xi_{N}} \right)} = {\prod\limits_{n = 1}^{N}\; {f^{(n)}\left( \xi_{n} \right)}}},$

defined on

^(N), to a plane.

In practice, each layer has discrete pixels with constant transmittancesrather than continuously-varying opacities. Thus, it is desirable totabulate the transmittance ƒ_(i) _(n) ^((n)) at each pixel i_(n) withinthe vector f^((n)). As shown in FIG. 4, each light field ray (x,v) canbe equivalently parameterized by the corresponding points ofintersection {ξ₁, ξ₂, . . . , ξ_(N)} with each layer. For a three-layerdisplay with discrete pixels, the intensity of the emitted light field{tilde over (l)}{ξ₁, ξ₂, ξ₃} is approximated by the productf_(i)g_(j)h_(k), where {i, j, k} denote the pixel indices nearest to thepoints of intersection {ξ₁, ξ₂, ξ₃}. With this parameterization,Equation 3 can be represented in discrete coordinates as a 3^(rd)-order,rank-1 tensor

, given by

=f∘g∘h, such that {tilde over (t)} _(ijk)=ƒ_(i) g _(j) h _(k),  (Eq. 5)

where ∘ is the vector outer product.

Only a subset of tensor elements {tilde over (t)}_(ijk) correspond tovalid light field rays; most tensor elements correspond to“non-physical” rays (i.e., ones that spontaneously change position ordirection after passing through a layer). To address this limitation oftensor representation, define a sparse, binary-valued weight tensor

such that the emitted light field tensor

is given by the following expression:

$\begin{matrix}{{= \cdot},{{{for}\mspace{14mu} w_{ijk}} = \left\{ \begin{matrix}1 & {{if}\mspace{14mu} \left\{ {i,j,k} \right\} \mspace{14mu} {gives}\mspace{14mu} a\mspace{14mu} {light}\mspace{14mu} {field}\mspace{14mu} {ray}} \\0 & {otherwise}\end{matrix} \right.}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$

where  is the Hadamard (elementwise) product.

Non-zero elements of

are close to the plane defined by Equation 4. Tensors provide sparse,memory-efficient representations for static N-layer displays; only thenon-zero elements of

are stored.

FIG. 5 is an overview of tensor displays. In FIG. 5: (Top left, i.e.,row 501 in column 511) A target light field for a teapot, rendered as5×5 views with a 20 degree field of view, is shown in the top left.(Second and Third Rows) Each column of the second and third rows showstwo examples of tensor display reconstructions of the scene, in adifferent implementation of this invention. Specifically, the second andthird rows 503, 505 show two reconstructions of: (i) a two-layer,12-frame display (column 513), (ii) a static three-layer display (column515), (iii) a three-layer, 12-frame tensor display (column 517), (iv) asingle-layer display using 12 frames and a directional backlight (column519), and (v) a two-layer display using 12 frames and a directionalbacklight (column 521). In columns 519, 521, the spatial backlightresolution is a quarter that of each layer. The reconstructions in FIG.5 show that time-multiplexing, as allowed by tensor displays,significantly reduces artifacts observed with the static three-layerconfiguration. The top row 501 (except for the left column 511) showsvisualizations of the light field, as restricted to the plane within thedisplay tensor

given by Equation 4. Specifically, these visualizations in the top row501 are shown for five examples in columns 511, 513, 515, 517, 519, 521,discussed above.

Time-Multiplexed Multilayer Displays: static multilayer displays havefinite degrees of freedom. Artifacts, resulting from limited depths offield and fields of view, persist in the emitted light field. Theseartifacts are typically observed as blur. These artifacts may bemitigated by increasing the degrees of freedom.

Increased degrees of freedom may be achieved by rapid temporalmodulation, such that the observer perceives the average of an M-framesequence.

Generalizing Equation 1, the emitted light field {tilde over (l)}(x,v)is given by

$\begin{matrix}{{\overset{\sim}{l}\left( {x,{v;N},M} \right)} = {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {\prod\limits_{n = 1}^{N}\; {f_{m}^{(n)}\left( {x + {\left( {d_{n}/d_{r}} \right)v}} \right)}}}}} & \left( {{Eq}.\mspace{14mu} 7} \right)\end{matrix}$

where ƒ_(m) ^((n))(ξ_(n)) is the transmittance at the point ξ_(n) oflayer n during frame m.

Let columns of the matrix F^((n))=[f₁ ^((n))f_(2(n)) . . . f_(M(n))]define the sequence of M masks displayed on layer n. For a three-layerdisplay, Equation 7 can be represented in discrete coordinates as a3^(rd)-order, rank-M tensor

given by

$\begin{matrix}{= {{〚{F,G,H}〛} \equiv {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {f_{m} \circ g_{m} \circ h_{m}}}}}} & \left( {{Eq}.\mspace{14mu} 8} \right)\end{matrix}$

where matrices enclosed by double square brackets correspond to the CPdecomposition (canonical polyadic decomposition) of a tensor into a sumof rank-1 tensors.

The CP decomposition is equivalent to CANDECOMP (canonicaldecomposition) and PARAFAC (parallel factors), with elements of thetensor

given by

${\overset{\sim}{t}}_{ijk} = {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {f_{im}g_{jm}{h_{km}.}}}}$

. For the general case with N light-attenuating layers and Mtime-multiplexed frames, the emitted light field can be represented asan N^(th)-order, rank-M tensor

=[[F⁽¹⁾, F⁽²⁾, . . . , F^((N))]].

Light field synthesis with time-multiplexed, multilayer displaysrequires decomposing a target light field l(x, v) into an M-framesequence of N transmittance functions ƒ_(m) ^((n))(ξ_(n)). This can beformulated as the following constrained nonlinear least squares problem:

$\begin{matrix}{{\underset{\{{f_{m}^{(n)}{(\xi_{n})}}\}}{\arg \mspace{11mu} \min}{\int_{v}{\int_{x}{\left( {{l\left( {x,v} \right)} - {\overset{\sim}{l}\left( {x,v} \right)}} \right)^{2}\ {x}\ {v}}}}},{{{for}\mspace{14mu} 0} \leq {f_{m}^{(n)}\left( \xi_{n} \right)} \leq 1}} & \left( {{Eq}.\mspace{14mu} 9} \right)\end{matrix}$

where {tilde over (l)}(x,v) is the emitted light field, given byEquation 7, and X and V denote the intervals [x_(min), x_(max)] and[v_(min), v_(max)].

The tensor representation discussed above provides an efficient meansfor solving Equation 9. Using this representation for a three-layerconfiguration with discrete coordinates, the objective function isexpressed as

$\begin{matrix}{{\underset{\{{F,G,H}\}}{\arg \mspace{11mu} \min}{{- {\cdot {〚{F,G,H}〛}}}}^{2}},{{{for}\mspace{14mu} 0} \leq F},G,{H \leq 1}} & \left( {{Eq}.\mspace{14mu} 10} \right)\end{matrix}$

where

is the target light field tensor, obtained by assigning the target lightfield l(x, v) to the plane defined by Equation 4, and

${}^{2} = {\sum\limits_{i = 1}^{I}\; {\sum\limits_{j = 1}^{J}\; {\sum\limits_{k = 1}^{K}\; x_{ijk}^{2}}}}$

is the squared tensor norm of χ.

This expression can be solved by applying weighted nonnegative tensorfactorization (NTF) and multiplicative update rules. For a three-layerdisplay, these update rules have the following forms:

F←F((W ₍₁₎ L ₍₁₎)(H⊙G))⋄((W ₍₁₎)(F(H⊙G)^(T)))(H⊙G))  (Eq. 11)

G←G((W ₍₂₎ L ₍₂₎)(H⊙F))⋄((W ₍₂₎(G(H⊙F)^(T)))(H⊙F))  (Eq. 12)

H←H((W ₍₃₎ L ₍₃₎)(G⊙F))⋄((W ₍₃₎(H(G⊙F)^(T)))(G⊙F))  (Eq. 13)

In these expressions, ⋄ is Hadamard (elementwise) division. Also, inthese expressions ⊙ is the Khatri-Rao product, defined for a pair ofmatrices Aε

^(I×K) and Bε

^(J×K), such that

A⊙B=[a ₁{circle around (x)}b₁ a ₂{circle around (x)}b₂ . . . a_(K){circle around (x)}b_(K)],  (Eq. 14)

where {circumflex over (x)} is the Kronecker product and a_(i) and b_(j)denote the i^(th) and j^(th) columns of A and B, respectively.

These update equations also make use of the tensor matricization(unfolding) operation, defined such that X_((n)) arranges the mode-nfibers of X to be columns of the resulting matrix.

For the general case with N light-attenuating layers and M frames,Equation 10 has the following form:

$\begin{matrix}{{\underset{\{ F^{(n)}\}}{\arg \mspace{11mu} \min}{{- \cdot}}^{2}},{{{for}\mspace{14mu} 0} \leq F^{(n)} \leq 1}} & \left( {{Eq}.\mspace{14mu} 15} \right)\end{matrix}$

where

=[[F⁽¹⁾, F⁽²⁾, . . . , F^((N))]].

Similarly, the update rules are generalized such that

F ^((n)) ←F ^((n))((W _((n)) L _((n)) F _(⊙) ^(n))⋄(W _((n))(F^((n))(F _(⊙) ^((n)))^(T)))F _(⊙) ^(n))  (Eq. 16)

where F_(⊙) ^(n) is defined by the following expression:

F _(⊙) ^(n) ≡F ^((N)) ⊙ . . . ⊙F ^((n+1)) ⊙F ^((n−1)) ⊙ . . . ⊙F⁽¹⁾  (Eq. 17)

4D light fields and 2D layers require vectorizing the 2D layertransmittances, giving a similar set of transmittance vectors f_(m)^((n)). Values are clamped to the feasible range after each iteration ofEquation 16.

According to principles of this invention, tensor representation allowsfor the decomposition of a target light field into a set oftime-multiplexed, light-attenuating layers. The multiplicative updaterules allow an efficient, GPU-based implementation that achievesinteractive refresh rates with multilayer LCDs.

As shown in the fourth column (517) of FIG. 5, time multiplexingsignificantly reduces artifacts observed with multilayer displays, asquantified by the peak signal-to-noise ratio (PSNR). Yet, such displaysare still restricted to relatively narrow fields of view (i.e., ≦20°).One way to expand the field of view would be to further increase therefresh rate. However, this solution may be precluded by the underlyingdisplay hardware.

In exemplary implementations of this invention, an alternate approachfor achieving wider fields of view is used: replacing conventionaluniform backlighting with time-multiplexed directional backlighting.

A directional backlight is equivalent to a low-resolution light fielddisplay. Consider a directional backlight that has significantly lowerspatial resolution, but equivalent angular resolution and field of view,as compared to the target light field l(x, v). In that case, it isdesirable to enhance the spatial resolution by covering a low-resolutionlight field display with an N-layer stack of light-attenuating layers.Generalizing Equation 7, the light field emitted by such a displayarchitecture is given by the following expression:

$\begin{matrix}{{\overset{\sim}{l}\left( {x,v} \right)} = {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {{b_{m}\left( {x,v} \right)}{\prod\limits_{n = 1}^{N}\; {f_{m}^{(n)}\left( {x + {\left( {d_{n}/d_{r}} \right)v}} \right)}}}}}} & \left( {{Eq}.\mspace{14mu} 18} \right)\end{matrix}$

where b_(m)(x, v) denotes the light field emitted by the backlightduring frame m.

Let B denote the discrete backlight light field, such that b_(as)corresponds to pixel s of view a. The backlight light field can beequivalently represented as a vector b, defined as follows.

b=[b ₁ ^(T) b ₂ ^(T) . . . b _(S) ^(T)]^(T), for b _(s) =[b _(1s) b_(2s) . . . b _(As)]^(T)  (Eq. 19)

Using this parameterization, Equation 18 can be represented in discretecoordinates as an N+1-order, rank-M tensor

, given by

$\begin{matrix}{= {\frac{1}{M}{b_{m} \circ f_{m}^{(1)} \circ f_{m}^{(2)} \circ \ldots \circ f_{m}^{(N)}}}} & \left( {{Eq}.\mspace{14mu} 20} \right)\end{matrix}$

where tensor element

${\overset{\sim}{t}}_{{ij}_{1}j_{2}\mspace{14mu} \ldots \mspace{14mu} j_{N}} = {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {b_{im}{\prod\limits_{n = 1}^{N}\; {f_{j_{n}m}^{(n)}.}}}}}$

Since Equations 8 and 20 are similar, NTF can also be applied tooptimize multilayer displays with directional backlighting.

As shown in FIG. 5, directional backlighting allows multilayer displaysto achieve wide fields of view, even with a single high-speed,light-attenuating layer. Tensor representation for multilayer displaysprovides a computationally-efficient optimization scheme encompassing awide variety of display architectures.

Tensor displays can exploit the additional degrees of freedom arisingfrom multiple layers and frames to achieve high-fidelity light fieldreconstructions.

The upper portion 605 of FIG. 6 shows reconstruction and decompositionresults for a tensor display that comprises three LCD layers and auniform backlight: The upper right side (columns 631 top, 632 top, 633top) show deconstruction results, the top left side 611 showsreconstruction results.

Objects close to the display appear sectioned across layers. Forexample, an object close to the display may map primarily to the frontlayer, with residual details assigned to other layers. Similarsectioning behaviors have been observed in the past with multilayer-onlydecompositions. Unlike these works, however, joint multilayer,multiframe decompositions produce additional time-varying,high-frequency patterns that appear across all layers and resemblecontent-adaptive parallax barriers.

The bottom portion 607 of FIG. 6 shows reconstruction and decompositionresults for a tensor display that comprises a front LCD layer and adirectional backlight. The bottom right side (columns 631, 632, 633)shows deconstruction results, the bottom left side 621 showsreconstruction results. The front LCD layer (shown in the bottom portion605 of columns 631, 632) displays view-independent portions of thescene, with flowing, slit-like patterns appearing around regions withview-dependent features. The directional backlight layer (shown in thebottom portion 605 of column 633) displays primarily view-dependentfeatures, such as objects extending from the physical display enclosure.

In FIG. 6, views 613, 623 show the position of the bunnies in the scene,viewed from the top.

Tensor display decompositions exhibit predictable structures, whosearrangement arise from the specific display configuration.Heuristically-defined methods can achieve similar fidelity with reducedcomputation.

The performance of an automultiscopic display can be quantified by itsdepth of field: an expression for the maximum spatial frequency ω_(ξ)_(max) that can be depicted in a plane oriented parallel to the screenand separated by a distance d_(o). This expression is derived using afrequency-domain analysis of the emitted light field {tilde over(l)}(x,v).

Taking the 2D Fourier transform of Equation 18 yields the followingexpression for the emitted light field spectrum {circumflex over(l)}(ω_(x), ω_(y)):

$\begin{matrix}{{\hat{l}\left( {\omega_{x},\omega_{v}} \right)} = {\frac{1}{M}{\sum\limits_{m = 1}^{M}\; {{{\hat{b}}_{m}\left( {\omega_{x},\omega_{v}} \right)}*\left\lbrack {\underset{n = 1}{\overset{N}{*}}{{\hat{f}}_{m}^{(n)}\left( \omega_{x} \right)}{\delta \left( {\omega_{v} - {\left( {d_{n}/d_{r}} \right)\omega_{x}}} \right)}} \right\rbrack}}}} & \left( {{Eq}.\mspace{14mu} 21} \right)\end{matrix}$

where ω_(x) and ω_(v) are the spatial and angular frequencies, * denotesconvolution, and the repeated convolution operator is defined as

$\begin{matrix}{{\underset{n = 1}{\overset{N}{*}}{\hat{f}}_{m}^{(n)}} \equiv {{{\hat{f}}_{m}^{(1)}\left( {\omega_{x},\omega_{v}} \right)}*\ldots*{{\hat{f}}_{m}^{(N)}\left( {\omega_{x},\omega_{v}} \right)}}} & \left( {{Eq}.\mspace{14mu} 22} \right)\end{matrix}$

For uniform backlighting, the backlight spectrum

{circumflex over (b)}_(m)(ω_(x), ω_(v))=δ(ω_(x), ω_(v)), the Dirac deltafunction.

The spectral support of a tensor display is the region of non-zerovalues in the emitted light field spectrum, for all possible layer masksand backlight illumination patterns. The spectral support for the lightfield reflected by a diffuse surface is the lineω_(v)=(d_(o)/d_(r))ω_(x).

Intersecting this line with the spectral support for a given displayprovides a geometric construction for the upper bound on the depth offield. For example, the emitted light field spectrum for a parallaxbarrier or integral imaging display is non-zero only for |ω_(x)|≦1/(2Δx)and |ω_(v)|≦1/(2Δv), where Δx and Δv are the spatial and angularsampling rates, respectively. In practice, the spatial sampling rate Δxis the spacing between barrier slits/pinholes or lenslets.

The geometric construction yields the following expression for the depthof field:

$\begin{matrix}{{\omega_{\xi_{\max}}\left( d_{o} \right)} = \left\{ \begin{matrix}\frac{1}{2\Delta \; x} & {{{{for}\mspace{14mu} {d_{o}}} \leq {d_{r}\left( \frac{\Delta \; x}{\Delta \; v} \right)}},} \\\frac{d_{r}}{2{d_{o}}\Delta \; v} & {{otherwise},}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 23} \right)\end{matrix}$

where Δv=(2d_(r)/A)tan(α/2) with A views and field of view α.

The geometric construction provides an upper bound on the depth of fieldfor any tensor display architecture. Consider a two-layer display withuniform backlighting, with the layers separated by a distance Δd andω₀=1/(2p) denoting the maximum spatial frequency for each layer withpixel pitch p. Equation 21 defines the light field spectrum, whered₁=−Δd/2 and d₂=Δd/2. A diamond-shaped region bounds the spectralsupport for any two-layer display. The spatial cutoff frequency ω_(ξ)_(max) is again found by intersecting the line ω_(v)=(d_(o)/d_(r))ω_(x)with the boundary of the spectral support, yielding the following upperbound on the depth of field for any two-layer display.

$\begin{matrix}{{\omega_{\xi_{\max}}\left( d_{o} \right)} = {\left( \frac{2\Delta \; d}{{\Delta \; d} + {2{d_{0}}}} \right)\omega_{0}}} & \left( {{Eq}.\mspace{14mu} 24} \right)\end{matrix}$

Using the previously described geometric construction, the depth offield for a three-layer display with uniform backlighting andequally-spaced layers is given by

$\begin{matrix}{{\omega_{\xi_{\max}}\left( d_{o} \right)} = \left\{ \begin{matrix}{\left( \frac{3\Delta \; d}{{\Delta \; d} + {d_{0}}} \right)\omega_{0}} & {{{{for}\mspace{14mu} {d_{o}}} \leq {2\Delta \; d}},} \\{\left( \frac{2\Delta \; d}{d_{o}} \right)\omega_{0}} & {{otherwise},}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 25} \right)\end{matrix}$

where Equation 21 is again applied to find the spectral support, withd₁=−Δd, d₂=0, and d₃=Δd.

The spectral support for a three-layer display exceeds that of a similarparallax barrier or integral imaging display, leading to increased depthof field.

Incorporating directional backlighting can significantly expand thefield of view. The depth of field for a single-layer display usingdirectional backlighting is obtained by a similar geometricconstruction.

Consider a directional backlight which implements a low-resolution lightfield display, such that {circumflex over (b)}_(m)(ω_(x), ω_(v)) hasnon-zero support for |ω_(x)|≦1/(2Δx) and |ω_(v)|≦1/(2Δv). This yieldsthe following depth of field expression:

$\begin{matrix}{{\omega_{\xi_{\max}}\left( d_{o} \right)} = \left\{ \begin{matrix}{\frac{1}{2\Delta \; x} + \omega_{0}} & {{{for}\mspace{14mu} {d_{o}}} \leq {d_{r}\left( \frac{\Delta \; x}{{\Delta \; v} + {2\Delta \; x\; \Delta \; v\; \omega_{0}}} \right)}} \\\frac{d_{r}}{2{d_{o}}\Delta \; v} & {{otherwise},}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 26} \right)\end{matrix}$

where ω₀ again denotes the spatial cutoff frequency for the layer.

The addition of a single light-attenuating layer significantly increasesthe spatial resolution for a conventional parallax barrier or integralimaging display, particularly near the display surface. However, farfrom the display, the depth of field is identical to these conventionalautomultiscopic displays.

Advantageously, tensor displays can achieve increased depth of field bycovering any low-resolution light field display with time-multiplexed,light-attenuating layers. In a prototype of this invention, theoptimization program uses continuously-varying layer transmittances.Alternately, the upper bound of the depth of field can be characterizedwith discrete pixels.

In some implementations of this invention: (a) static andtime-multiplexed tensor displays have identical spectral supports (i.e.,averaging over an M-frame sequence does not alter the support viaEquation 21); yet (b) time multiplexing significantly reduces artifacts.Without being limited by theory, the reduced artifacts may beattributable, at least in part, to additional degrees of freedom allowedwith time multiplexing. While the upper bound of the depth of field maybe identical, in practice it cannot be achieved with static methods,motivating tensor displays for joint multilayer, multiframedecompositions capable of approaching the upper bound.

An important benefit of tensor displays is to open a design trade spacenot accessible to conventional automultiscopic displays. Conventionalmultilayer-only or multiframe-only decompositions require many layers orprohibitively high frame rates, limiting their practicality usingcurrent LCD technology. However, with joint multilayer, multiframedecompositions, display designers can explore the interdependence of thenumber of layers, the number of frames, and the image brightness.

In exemplary implementations, tensor displays use fewer layers andframes achieve higher-fidelity reconstructions than conventionalmethods, in a manner supported by current LCD technology. Tensordisplays can achieve wide fields of view, as required for multiviewerscenarios.

Consider a fixed set of uniformly-spaced viewpoints during optimization.Providing closely-spaced target views sufficiently constrains thedecompositions so minimal artifacts are perceived at intermediateviewpoints.

In some implementations, it is desirable to maximize image fidelity(e.g., PSNR) as a function of device complexity (i.e., the number oflayers and frames). Increasing the number of frames allows the number oflayers to be decreased (for a given PSNR). Image fidelity also dependson the brightness scale βε[0,1] applied to the target light field.Modifying Equation 15 yields the following objective function supportinga trade-off between image brightness and fidelity.

$\begin{matrix}{{\underset{\{ F^{(n)}\}}{\arg \mspace{11mu} \min}{{ - \cdot}}^{2}},{{{for}\mspace{14mu} 0} \leq F^{(n)} \leq 1}} & \left( {{Eq}.\mspace{14mu} 27} \right)\end{matrix}$

Decreasing brightness generally yields higher-fidelity reconstructionsfor the same number of layers and frames.

For example, consider the trade space for multilayer displays withbrightness β=0.2 m, in a prototype of this invention. In this example:(a) static decompositions (i.e., M=1) cannot exceed 30 dB, even with asmany as eight layers; (b) to achieve 40 dB with eight layers, two framesare required; (c) there is a trade-off between layer complexity andrefresh rate along the 40 dB curve; and (d) using six frames, only threelayers are required, with more frames providing marginal benefits. Thus,with tensor displays, high-speed displays may be used to reduce devicecomplexity, minimizing the number of layers to achieve a certain imagefidelity.

Adding a directional backlight alters the design trade space. Forexample, consider a prototype of this invention, in which a directionalbacklight has 47×29 lenslets. In this directional backlight example: (a)two frames are still required to reach 40 dB using eight layers; but (b)only a single layer is required using eight frames. In this example, thedirectional backlight effectively reduces the number of required layersby one. This underscores the practical benefits of the tensor displayframework, which can employ multilayer decompositions,time-multiplexing, and directional backlighting together.

In other applications of a prototype of this invention: (i) for threelayers and a uniform backlight, four frames are required to achieve 40dB, and with additional frames, brightness can be significantlyincreased (up to β≈0.6); and (ii) for single layer and a directionalbacklight, a minimum of eight frames are required to achieve 40 dB. (Ifthe directional backlight itself includes an LCD layer, then a singleLCD layer and directional backlight actually comprise two LCD layers).

Conventional automultiscopic displays, including parallax barriers andintegral imaging, exhibit a set of periodically-repeating viewing zones.In contrast, recent (prior art) computationally-optimized multilayer andmultiframe displays generally (i) exhibit a set of non-repeating viewingzones, and (ii) yield extended depths of field, greater resolution, andincreased brightness. However, these (prior art)computationally-optimized multilayer and multiframe displays typicallyhave, for any single viewer, a limited field of view per of α≦20°.

Tensor displays can support wider fields of view, while retaining thebenefits of computational optimization. In a prototype of thisinvention, a field of view of α=50°×20° is achieved, for a light fieldwith 9×3 views, using either five layers and uniform backlighting or asingle layer and directional backlighting. Prior art multilayer-only andmultiframe-only decompositions lack sufficient degrees of freedom toachieve high-PSNR reconstructions for these scenarios.

A prototype of this invention comprises a reconfigurable tensor displaycapable of implementing two-layer and three-layer architectures withuniform or directional backlighting. The layers are constructed usingthree modified Viewsonic® VX2268wm 120 Hz LCD panels. The front and rearpolarizing films are removed from the front two LCDs, and the stack isinterleaved with alternating crossed linear polarizers. Aluminumbrackets added to the rear panel allow lenslet arrays to be affixed foroperation as a directional backlight. A rectangular lenslet array isapproximated using two crossed lenticular sheets, purchased from MicroLens Technology, Inc. The corrugated surfaces of the sheets are held indirect contact, minimizing astigmatic aberrations. The directionalbacklight supports varying spatio-angular resolution trade-offs using10, 15, and 20 lenses per inch (LPI) lenticular sheets. In directionalbacklighting modes, an additional polarizing film is placed after thelenslet arrays, restoring the linear polarization state before raysimpinge on the next LCD in the stack.

This prototype employs offline and online solvers based on Equation 16.Computation is divided between CPUs (central processing units) for theoffline solver, and GPUs (graphical processing units) for the onlinesolver. The offline solver is run on an Intel Core® i5 workstation with10 GB of RAM. The online solver is run on an Intel Core i7 workstationwith 6 GB of RAM and an external Nvidia® QuadroPlex 7000 graphics unitcontaining two Quadro® GPUs and a G-Sync card. This provides fourframe-synchronous DVI (Digital Visual Interface) outputs capable ofdriving the LCDs at 120 Hz.

In this prototype, target light fields are rendered using POV-Ray(Persistence of Vision Raytracer program) or, for interactiveapplications, using OpenGL (Open Graphics Language). Rendered lightfields have a spatial resolution of 840×525 pixels (i.e., half theresolution of LCDs used in this prototype) and an angular resolution of5×5 views.

This prototype employs nonnegative tensor factorization (NTF) using themultiplicative update rules described above. An offline, Matlab®-basedsolver is used for simulations. Decomposing a target light field into asix-frame sequence for three layers takes approximately 30 minutes using50 updates. Color channels are processed independently. An online,GPU-accelerated solver is implemented in OpenGL and Cg (C for Graphics).In some applications, the update rules are cast as additive combinationsof the logarithms of the layer transmittances. Using thisrepresentation, the update rules are mapped to standard operations ofthe graphics pipeline, including projective texture mapping,accumulation buffers, floating point framebuffers, and perspectiverendering. These operations are not only computationally efficient, butalso memory-efficient, as only the non-zero tensor elements need to bestored and processed. For interactive applications, temporal coherencebetween decompositions may be exploited, seeding each frame with theprior result. Portions of the pseudocode used for the GPU-acceleratedsolver is set forth in the Pseudocode section below.

In this prototype, separate threads are used to decouple thedecomposition from the display routines. Decompositions are evaluated inan asynchronous thread, updating layer patterns as they becomeavailable. This ensures that all display layers can be continuouslyrefreshed at 120 Hz, without waiting for updated decompositions. Thisprototype can achieve up to 10 multiplicative updates per second for asmany as 12 frames. Light fields with reduced spatial or angularresolution can be decomposed and displayed at interactive refresh rates,as shown in the supplementary video.

In this prototype, the multiplicative updates constitute an alternatingleast squares solution to the nonlinear tensor factorization problem,employing steepest descent with a fixed step length. While this approachtypically exhibits slow convergence using a CPU-based implementation,each update is efficiently computed using the GPU-acceleratedimplementation. To support interactive applications, temporal coherencebetween decompositions can be exploited, seeding each frame with theprior decomposition. For static scenes, seeding results in one updateper frame. For interactive applications, seeding introduces motion blur.Given sufficient computational resources, blur can be eliminated byusing multiple updates per frame.

This prototype is reconfigurable, as noted above. In one configuration,this prototype comprises a three-layer LCD with uniform backlighting.

In this three-layer, uniform backlighting configuration: Acrylic spacersseparated each panel by Δd=4.0 cm. The target light field was renderedwith a field of view of α=20°×20° and brightness β=0.2 (see Section 4.2

Experiments with the prototype provide insights into practicalengineering issues. Accurate mechanical alignment is desirable.Decomposed layers exhibit high-frequency patterns; it is desirable thatthese patterns be properly aligned. Accurate alignment is ensured bydisplaying perspective images of a crosshair array on each layer. Acamera is placed at the desired viewer position (e.g., directly in frontof the display at a distance of 2 m) and the patterns were shifted untilalignment was obtained. Radiometric calibration is desirable, includingmeasuring the black levels and gamma values. The former are incorporatedas constraints in the update rules, while the latter are addressed byapplying gamma correction at runtime. Without being limited by theory,remaining variations in color and intensity may be attributable todifferences in the LCD color gamut, color filter cross-talk, moire dueto stacking multiple layers, and angular color variation common tohigh-speed LCDs.

In a second configuration, this prototype comprises a single LCD with adirectional backlight. The backlight uses crossed 10 LPI lenticularsheets, yielding a field of view of α=48′×48′ and backlight resolutionof 187×117 lenslets. The front LCD is separated by Δd=8.5 mm from themiddle of the lenticular sheets. Remaining system parameters areidentical to the three-layer prototype. The crossed lenticular sheetsproduce strong absorption along lens boundaries. In a commercialimplementation, lenslet arrays can be manufactured with minimalabsorption. Alternatively, edge-lit directional backlighting caneliminate this artifact. As shown in FIG. 7, adding an LCD in front of alow-resolution directional backlight increases the spatial resolutionfor virtual objects appearing on the display surface (e.g., the logo719) and for objects extending in depth (e.g., the fish tail on theright 721). Resolution can be enhanced at the display surface withouttime multiplexing. However, time multiplexing allows resolutionenhancement for extended scenes. Such time multiplexing is facilitatedby a tensor framework.

FIG. 7 shows an original light field for a scene (701, 711, 713), imagesof the scene produced by integral imaging (703, 715, 717), and images ofthe scene produced by a tensor display (707, 719, 721). While integralimaging, implemented in this example with a lenslet array affixed to anLCD, achieves a convincing 3D effect, spatial resolution issignificantly reduced (703, 715, 721). Adding an LCD in front of thelow-resolution backlight and exploiting temporal multiplexing using atensor framework increases the spatial resolution, not only on thephysical layers, but also outside the hardware enclosure (707 719, 721).Views 705, 709 show the position of the fish in the scene, viewed fromthe top.

Tensor displays open a large design trade space that was inaccessibleusing prior automultiscopic displays. With a tensor display framework,designers can maximize image fidelity, brightness, and field of view,depending on the number of layers and maximum refresh rate allowed bythe design constraints and display technology, respectively.

In some cases, this invention is implemented with a single LCD with adirectional backlight. This design has many advantages. Such displayssupport a wide field of view with relatively few frames (i.e., as few asthree). Thus, provided with 180 Hz LCDs, this design can achieve a thinform factor, wide field of view, bright automultiscopic display with aneffective refresh rate of 60 Hz.

In some cases, this invention employs joint multilayer, multiframedecompositions with uniform backlighting. These decompositions can be aneffective tool for optimizing multilayer displays with uniformbacklighting. Such displays with uniform backlighting can have the addedbenefit of a tunable field of view. This allows viewing zones to adaptto the location of viewers. (In contrast, in one of the prototypes ofthis invention, directional backlighting has a fixed field of view).

A prototype of this invention exhibits several limitations inherent tolayered architectures, including moire, color-channel crosstalk,interreflections, misalignment, and dimming due to layered color filterarrays. Many of these issues can be resolved with additional opticalengineering. Moire, interreflections, and misalignment can be mitigatedusing holographic diffusers, antireflective coatings, and rigidenclosures, respectively. A direct solution to crosstalk is to alter thetransmission profiles of the color filters; however, this approach willfurther decrease brightness. Instead, field sequential color can beapplied (i.e., using a backlight that sequentially strobes each color),albeit by placing additional demands on the refresh rate.

In some implementations of this invention, color filters are not usedfor each layer. In that case, decompositions are performed assumingmonochromatic panels interspersed with a few color filters.

The weight tensor applied in Equation 16 allows decompositions to betuned to the positions of viewers. Head or eye tracking may be employed,and the weight matrix can be altered to only project automultiscopicimagery aligned to each viewer. Between viewers, the emitted light fieldcan be unconstrained, allowing for higher-fidelity, brighter imagery.

The image formation model, given by Equation 18, can be generalized. Forexample, the model can be generalized to apply to time-multiplexed,light-attenuating layers over a uniform light source, with one lensletarray between the first and second layers.

In some implementations of this invention, at least some of the layerscomprise both light-attenuating and light-emitting materials.Optionally, refractive elements can be placed at any point (e.g., aFresnel lens in front of the display to extend the depth of field).

In some implementations of this invention, additional views are used tocreate accommodation and convergence cues. To support more views,higher-speed displays can be employed. For example, tensor displays canbe implemented with digital microshutters (DMS), capable of achieving1,440 Hz refresh rates, allowing 24 frames with an effective refreshrate of 60 Hz.

In a prototype of this invention, least-squares optimization isemployed. Alternately, perceptual error metrics may be used in tensordisplays. Advantageously, perceptual error metrics allow furtherreductions in complexity (i.e., fewer layers and frames). In some cases,perceptual error metrics involve nonlinear objectives and use modifiedoptimization schemes.

Tensor displays can bring together the advantages of multilayer panels,high refresh rates, and directional backlighting. In exemplaryimplementations of this invention, tensor displays comprisecomputational displays, wherein the display architecture and encodingalgorithm are jointly optimized to maximize optical and computationalefficiency.

This invention can be implemented as a single LCD with a directionalbacklight. This design achieves a wide field of view and large depth offield with a thin form factor using efficient multiplicative updates.

Tensor displays may be implemented in many different ways. Preferably,in tensor displays, multiple light-attenuating optical elements arecombined in a way such that each ray in a target light field intersectseach optical element at most once. Light-attenuating elements can bearranged in layers comprising any of the following: angularly-invariantspatial light modulators, purely directional modulators, andspatio-angular modulators. A low-resolution light field backlight, forinstance, implemented by a lenslet array on top of an LCD, is one typeof spatio-angular modulator.

In exemplary implementations of this invention, the tensor space spannedby a tensor display with N optical elements, such as layers, is ofdimension N. The light field only occupies a low-dimensional manifoldwithin the tensor space. The shape of the manifold depends on aparticular tensor display configuration (e.g., a three-layer display ora dual-layer configuration with an additional directional backlight). Aweighted nonnegative tensor decomposition has non-zero values only onthe low-dimensional manifold created by the light field in tensor space.

FIG. 8 illustrates an original light field with 3×3 views (row 801) anddecompositions (rows 803, 805, 807, 809, 811, 813) for a high-resolutionlayer directly on top of a purely directional backlight. This kind ofbacklight corresponds to a single large lens directly behind an LCD withanother spatial light modulator (SLM) mounted at the focal length of thelens; the secondary SLM has a resolution of 3×3, corresponding to theangular resolution of the light field. Decompositions for bothhigh-resolution LCD and low resolution angular backlight are shown forthree time-multiplexed frames (rows 803, 805), six frames (rows 807,809), and nine frames (rows 811, 813). The brightness for alldecompositions is scaled by the inverse of the number of frames. As seenin the lower two rows (rows 811, 813), NTF converges toward the obvioussolution: turning on each direction of the backlight sequentially overtime with the LCD showing the corresponding view of the light field.NTF, however, generalizes the factorization problem to an arbitrarynumber of frames and different brightness tradeoffs. For the case ofrank-deficient decompositions (rows 803, 805, 807, 809), the views andcorresponding backlight directions are automatically grouped into theset of structurally similar views that result in the optimal imagequality.

In some implementations of this invention, a low-resolution directionalbacklight combined with a high-resolution layer, such as an LCD, canachieve high image quality by temporally multiplexing only a few frames.

FIG. 9 shows NTF-based multilayer decompositions of a scene. In FIG. 9,the leftmost column comprises reconstructions of a scene using (i) atomographic solution (901, left), (ii) NTF, with one frame (903, left),and (iii) NTF, with three frames (905, left). Columns 907, 909, 911,913, 915 show decompositions for Layer 1, Layer 2, Layer 3, Layer 4,Layer 5.

As shown in FIG. 9: A tomographic five-layer decomposition (row 901) isintuitive, because it acts similar to a 3D geometry slicing operator fordiffuse objects inside the physical display enclosure. For globalillumination effects and objects outside the layers, however, thedecompositions are more complicated. A nonnegative tensor factorizationfor the same optical configuration, without any time multiplexing, isshown in row 903. The decompositions show a close similarity to thetomographic solution. A difference between the two rows (901, 903) isthat the tomographic solution is computed in log-space, resulting in alinear problem which can be solved efficiently, but with biased errors.As seen in column 907, specular highlights are slightly blurred andartifacts resulting from the parallax between different viewpoints aremore pronounced. By adding temporal multiplexing, as shown in the lowertwo rows 905, the achieved quality can be significantly improved. Thedecompositions themselves still resemble a slicing operator in the lowerfrequencies, but what is perceived as temporally-varying high-frequencynoise (lower two rows 905 in columns 909, 911, 913, 915) actuallycontains the information necessary to improve the resulting 3D imagequality. Note that any multiframe decompositions computed with NTFrepresent a tradeoff between PSNR and brightness; the latter is enhancedfor the simulated reconstruction in row 905 (left).

As shown in FIG. 9, the low spatial frequencies in NTF-decomposed layersare comparable to the tomographic solution. This acts similarly to a 3Dgeometry slicing operator for Lambertian objects on the layers.Multiframe decompositions computed with the tensor frameworkadditionally contain high-frequency variations in image regionsexhibiting motion parallax. Although these high-frequencies could beperceived as noise, they actually contain the information that increasesthe 3D image quality for temporally-multiplexed tensor displays.Multilayer decompositions computed with nonnegative tensor factorizationare structurally similar to the tomographic case if no temporalmultiplexing is used, but combine the advantages of multiple layers withtemporal multiplexing for all other cases.

In a prototype of this invention (with three LCD layers and a uniformbacklight), a minimum of 50 iterations is usually needed to ensure highimage fidelity, but about 6 to 12 time-multiplexed frames achieve a highimage quality even for a challenging scene exhibiting a large depth offield. In this prototype, light fields with uncorrelated views, such asArabic numerals, can be successfully synthesized using the proposedlow-rank tensor factorization.

In a test of this prototype (with three LCD layers and a uniformbacklight): (i) low image quality can be achieved due to a large depthof field; (ii) low-rank approximations using 6 and 12, respectively,time-multiplexed frames create a visually appealing approximation of thelight field; and (iii) higher-rank factorizations do not improve imagequality significantly, demonstrating that light field tensors areinherently of low rank.

FIG. 10 shows multilayer decompositions of a scene, using a directionalbacklight. A directional backlight adds more degrees of freedom thatincrease the field of view and depth of field of the tensor display.

Specifically, FIG. 10 shows decomposition for dual-layer displaycontaining a purely angular backlight behind the rear layer. Theoriginal light field has 4×4 views within a field of view of 30°. Two ofthe original views are shown on the upper left 1001, 1003 withcorresponding reconstructions 1005, 1007 next to them. This data setrepresents a rank-16 light field, which is decomposed using 10 frames.The two layers are separated by a distance that corresponds to theseparation distance for an equivalent parallax barrier display. Eachframe is shown for the front layer (row 1009), for the rear layer (row1011), and for the angular backlight (row 1013). The layerdecompositions resemble what NMF produces for dual-layer setups, butadds an angular backlight for improved depth of field and field of view.

DEFINITIONS AND CLARIFICATIONS

Here are a few definitions and clarifications. As used herein:

The terms “a” and “an”, when modifying a noun, do not imply that onlyone of the noun exists. For example, if “a” ball exists, this does notimply that only one ball exists.

A display is “automultiscopic” if it produces a 3D image that can beperceived by a human not wearing glasses or other optical apparatus. The3D image produced by an automultiscopic display, when viewed by a humannot wearing glasses or other optical devices: (i) includes multipleviews, the view seen depending on the angle at which the image isviewed, (ii) exhibits binocular disparity; and (iii) exhibits motionparallax in both horizontal and vertical directions.

A “backlight” provides illumination for one or more transmissivecomponents of a display device. The transmissive components areoptically closer to a viewer, compared to the backlight, which isoptically further from the viewer.

The term “comprise” (and grammatical variations thereof) shall beconstrued broadly, as if followed by “without limitation”. If Acomprises B, then A includes B and may include other things.

A backlight is “directional” if it is configured to output light at anangle, relative to a display surface of the backlight, that can vary atdifferent times. Here are some examples of “directional” backlights: (i)In a simple implementation, a directional backlight may comprise anarray of light-emitting devices behind a lens, where the differentlight-emitting devices are configured to be turned on one at a time,causing light to exit the front of the lens at an angle that varies overtime. (ii) In some cases, a directional backlight may include a layercomprising more than one pixel, the angle of light outputted by eachpixel in that layer being separately controllable. (iii) In some cases,a directional backlight may output light at multiple angles at the sametime. (iv) In some cases, a directional backlight may include a layercomprising a spatial attenuator, the attenuation caused by each pixel inthat layer being separately controllable.

The term “e.g.” means including without limitation.

To minimize the “error” between two things is to minimize a measure of(or based on) the difference between the two things. For example, asolution to a least squares problem may minimize the error between anactual and a target light field, in a least squared sense.

The fact that an “example” or multiple examples of something are givendoes not imply that they are the only instances of that thing. Anexample (or a group of examples) is merely a non-exhaustive andnon-limiting illustration.

Unless the context clearly indicates otherwise: (1) a phrase thatincludes “a first” thing and “a second” thing does not imply an order ofthe two things (or that there are only two of the things); and (2) sucha phrase is simply a way of identifying the two things, respectively, sothat they each can be referred to later with specificity (e.g., byreferring to “the first” thing and “the second” thing later). Forexample, unless the context clearly indicates otherwise, if an equationhas a first term and a second term, then the equation may (or may not)have more than two terms, and the first term may occur before or afterthe second term in the equation. A phrase that includes “a third” thing,a “fourth” thing and so on shall be construed in like manner.

The “flicker fusion frequency”, as used herein, means 30 Hz.

In the context of a display device (and components of the device),“front” is optically closer to a viewer, and “rear” is optically furtherfrom the viewer, when the viewer is viewing a display produced by thedevice during normal operation of the device. The “front” and “rear” ofa display device continue to be the front and rear, even when no vieweris present. Similar terms, such as “behind”, shall be construed in likemanner.

The terms “horizontal” and “vertical” shall be construed broadly. Forexample, “horizontal” and “vertical” may refer to two arbitrarily chosencoordinate axes in a Euclidian two dimensional space.

The term “include” (and grammatical variations thereof) shall beconstrued broadly, as if followed by “without limitation”.

“Intensity” shall be construed broadly to include any measure of orrelated to intensity, energy or power. For example, the “intensity” oflight includes any of the following measures: irradiance, spectralirradiance, radiant energy, radiant flux, spectral power, radiantintensity, spectral intensity, radiance, spectral radiance, radiantexitance, radiant emittance, spectral radiant exitance, spectral radiantemittance, radiosity, radiant exposure and radiant energy density.

As used herein, the “number” of spatially addressable, light attenuatinglayers in a display device does not count any such layers that are partof a directional backlight that illuminates the display. For example, ifthe only spatially addressable, light attenuating layers in a displaydevice are (1) a front LCD layer and (2) an LCD layer that is part of adirectional backlight, then the “number” of such spatially addressable,light attenuating layers is treated as one.

The term “or” is an inclusive disjunctive. For example “A or B” is trueif A is true, or B is true, or both A or B are true.

“NTF” means nonnegative tensor factorization. The “order” (or“dimension”) of a tensor is the minimum number of indicia needed touniquely identify a component of the tensor.

A parenthesis is simply to make text easier to read, by indicating agrouping of words. A parenthesis does not mean that the parentheticalmaterial is optional or can be ignored.

Persistence of vision is not a “perception error metric”, as usedherein.

To vary something “per pixel” means to vary it at respective pixels.Something may vary “per pixel” even if it varies only at some, but notall, of the pixels in a set of pixels. Similar terms (e.g. “on a perpixel basis”) shall be construed in like manner.

A “pixel” includes the smallest addressable element in a display device.For example, a light-transmitting or light-emitting display device mayhave pixels.

The “rank” of a tensor

is the minimum number of simple tensors with which it is possible toexpress

as a sum.

A “simple tensor” can be completely factorized into vectors.

A “single-layer” display using directional backlighting can beimplemented in multiple ways. For example, a “single-layer” displayusing directional backlighting may comprise a single LCD layer and apurely angular backlight. Or, for example, a “single-layer” displayusing directional backlighting may comprise both (1) a front LCD layerand (2) a directional backlight, where the directional backlight itselfincludes another LCD layer. The phrase “single-layer” is used toindicate that there is only one spatially addressable light attenuatinglayer, not counting any such layer in the directional backlight. Similarphrases (however worded) regarding a single layer (or single LCD) and adirectional backlight shall be construed in like manner.

The term “sparse” shall be construed broadly. For example, in aconventional context, a tensor is “sparse” if a majority of the tensorcomponents are zero. This allows significant data compression, zerosbeing insignificant data that is not stored. Of course, in anunconventional binary system, a conventional zero may be replaced withanother value. More generally, a tensor is “sparse” if it functions in amanner equivalent to a conventional sparse tensor. For example, a tensoris “sparse” if a majority of its components are a value that is treatedas insignificant data for compression purposes.

A display layer is “time-multiplexed” if it is configured to display asequence of frames at a rate equal to or faster than the flicker fusionfrequency. For example, a display layer that is configured to display asequence of frames at a rate of 60 Hz (60 frames per second) istime-multiplexed.

A “tensor display” is an automultiscopic display device wherein: (1) thedevice includes one or more spatially addressable, light attenuatinglayers; (2) the device includes a controller, which is configured toperform calculations to control the device; and (3) the calculationsinvolve using weighted NTF.

A backlight is “uniform” if it is not directional.

Notation

Here are examples of mathematical notation (e.g., font, capitalization,and math symbols) used herein:

α is a scalar;

a is a vector;

A is a matrix;

χ is a tensor;

χ_((i)) is matricization (unfolding) of tensor χ along mode i;

χx_(i)A=AX_((i)) is a tensor-matrix product along mode i;

a∘b is a vector outer product;

AB is a Hadamard matrix product (elementwise product);

A⋄B is Hadamard matrix division (elementwise division);

A{circumflex over (x)}B is a Kronecker product of two matrices A, B;

A_({circumflex over (x)})=A^((N)){circumflex over (x)} . . . {circumflexover (x)}A^((I)) is a Kronecker product of N matrices A^((N)), . . . ,A^((I));

A_({circle around (x)}) ^(n)=A^((N)){circumflex over (x)} . . .{circumflex over (x)}A^((n+1)){circumflex over (x)}A^((n−1)){circumflexover (x)} . . . {circumflex over (x)}A^((I)) is a Kronecker product ofN−1 matrices A^((N)), . . . , A^((I)), skipping A^((n));

A⊙B is a Khatri-Rao product of two matrices;

A_(⊙)=A^((N))⊙ . . . ⊙A⁽¹⁾ is a Khatri-Rao product of

N matrices A^((N)), . . . , A⁽¹⁾; and

A_(⊙) ^(n)=A^((N))⊙ . . . ⊙A^((n+1)) ⊙A^((n−1)) ⊙ . . . ⊙A⁽¹⁾ is aKhatri-Rao product of N−1 matrices A^((N)), . . . , A⁽¹⁾, skippingA^((n)).

Variations:

This invention may be implemented in many different ways. Here are somenon-limiting examples.

This invention may be implemented as a method comprising, incombination: (a) using a backlight to provide light to a display device,which display device includes one or more spatially addressable, lightattenuating layers; (b) using the layers to display a temporal sequenceof frames; and (c) using one or more processors (i) to perform anoptimization calculation to compute, for each respective frame in thesequence and each respective layer in the one or more layers,attenuation of the light at respective pixels of the respective layer;and (ii) to output control signals to control the attenuation; wherein(I) the optimization calculation includes at least one mathematicaloperation an N^(th)-order, rank-M tensor, where M is equal to the numberof frames in the sequence, and N is equal to the number of the layers,if the backlight is uniform, and N is equal to the number of the layersplus one, if the backlight is directional, (II) the optimizationcalculation includes applying a weighted nonnegative tensorfactorization, and (III) either (A) the backlight is uniform and thenumber of the layers is at least three or (B) the backlight isdirectional and the number of the layers is at least one. Furthermore:(1) the backlight may be directional; (2) the backlight may bedirectional and the number of the layers may be at least two; (3) thebacklight may be uniform and the number of the layers may be at leastthree; (4) the tensor may be sparse. The display device may beconfigured to produce an automultiscopic display. The automultiscopicdisplay may have one or more fields of view; the display device may havea front display surface; each respective field of view, out of the oneor more fields of view, may be centered about a viewing axis; and themethod may further comprise dynamically varying the viewing axis of eachfield of view, respectively, including to orientations that are notnormal to the front display surface, and tracking gaze or head positionof a human user of the display device.

This invention may be implemented as apparatus comprising, incombination: (a) a display device, which display device includes one ormore spatially addressable, light attenuating layers, which layers areconfigured to display a temporal sequence of frames; (b) a backlight,the backlight being configured to provide light to the display device;and (c) one or more processors, the one or more processors beingconfigured (i) to perform an optimization calculation to compute, foreach respective frame in the sequence and each respective layer in theone or more layers, attenuation of the light at respective pixels of therespective layer; and (ii) to output control signals to control theattenuation; wherein (I) the backlight is directional and the number ofthe layers is at least one, (II) the optimization calculation includesat least one mathematical operation on an rank-M tensor, M being equalto the number of frames in the sequence, and (III) the optimizationcalculation includes applying a weighted nonnegative tensorfactorization. The backlight may be directional and the tensor may havean order equal to N+1, where N is the number of the layers. Thebacklight may comprise a lens and a spatially addressable lightmodulating layer; the lens may have a focal length; and the lightmodulating layer may be positioned at a distance from the lens, whichdistance is equal to the focal length. The backlight may be directionaland the number of the layers may be equal to at least two. The displaydevice may be configured to produce an automultiscopic display. Thetensor may be sparse. The optimization calculation may optimize based atleast in part on perception error metrics. The optimization calculationmay calculate a set of per pixel attenuations, which set minimizes errorbetween a light field transmitted from the display device and a lightfield that would be created by a target 3D scene. At least one of thelayers may comprise both (i) optical elements configured to transmitlight and (ii) optical elements configured to emit light.

This invention may be implemented as apparatus comprising, incombination: (a) a display device, which display device includes one ormore spatially addressable, light attenuating layers, the layers beingconfigured to display a temporal sequence of frames; (b) a backlight,the backlight being configured to provide light to the display device;and (c) one or more processors, the one or more processors beingconfigured (i) to perform an optimization calculation to compute, foreach respective frame in the sequence and each respective layer in theone or more layers, attenuation of light at respective pixels of therespective layer; and (ii) to output control signals to control theattenuation; wherein (I) the backlight is uniform, (II) the optimizationcalculation includes at least one mathematical operation on anN^(th)-order, rank-M tensor, M being equal to the number of frames inthe sequence and N being equal to the number of the layers, (III) theoptimization calculation includes applying a weighted nonnegative tensorfactorization, and (IV) the number of the layers is at least three. Thedisplay device may be configured to produce an automultiscopic display,which display concurrently has one or more fields of view; the displaydevice may have a front display surface; each respective field of view,out of the one or more fields of view, may be centered about a viewingaxis; and (d) the display device may be configured to dynamically varythe viewing axis of each field of view, respectively, including toorientations that are not normal to the front display surface. Theapparatus may be configured to track gaze or head position of a humanuser of the display device. The tensor may be sparse.

It is to be understood that the methods and apparatus that are describedabove and below are merely illustrative applications of the principlesof the invention. Numerous modifications may be made by those skilled inthe art without departing from the scope of the invention.

Pseudocode:

The following pseudocode documents a GPU-based implementation ofnonnegative tensor factorization for tensor displays. all underlyingoperators for this particular application map well to functions of thefixed graphics pipeline. NTF is implemented using OpenGL and a set of CGshaders.

The pseudocode is designed for a tensor display consists of Llight-attenuating layers, each displaying F frames in rapid succession.An optional, low-resolution directional backlight is also supported.Both the original light field and the backlight are assumed to consistof V different views. As the display of the decomposed layers is atime-critical operation, requiring a frame rate that matches the monitorrefresh rate, it is implemented in a different thread than thedecomposition, which can be run at a lower frame rate. The separatedecomposition thread updates the light field at interactive frame ratesand decomposes it into a set of L layers, each with F different timeframes, and an additional directional backlight with V views and F timeframes.

The pseudocode documents the main display loop (i.e., the algorithmtitled NTF—Main Display Routines) for synchronized rendering oftemporally-multiplexed layers and the backlight with monitor refreshrates. This implementation assumes that calibrated interlacing masks areavailable for each view of the light field. These masks are multipliedby the corresponding rendered view and added together to generate aninterlaced image to be displayed behind a lenslet array.

Decomposition routines (e.g., the algorithm titled NTF—Content-UpdatingThread) are also documented, implementing weighted nonnegative tensorfactorization.

Algorithm: NTF - Main Display Routines variables FBO_LAYERS[ L ][ F ],FBO_BACKLIGHT[ V ][ F ], INTERLACING_MASKS[ V ], f =0, UseBacklight=truefunction mainDisplayLoop // draw layers of current frame for all layersl set viewport for l activate FBO_LAYERS[l][f] draw textured 2D quad end// draw backlight of current frame  If bUseBacklight set viewport forbacklight activate accumulation buffer for all views v activateCG_SHADER_MULTIPLY_TWO_TEXTURES  (INTERLACING_MASK[v],FBO_BACKLIGHT[v][f]) draw textured 2D quad end deactivate accumulationbuffer end // cycle through frames  f = (f < F) ? f +1 : 0; endAlgorithm NTF - Content-Updating Thread variables FBO_LF[V],FBO_LF_REC[V], FBO_LF_TMP[V],  FBO_LAYER_TMP[2], FBO_TMP\\ functionthreadDisplayLoop( ) // draw light field for all light field views vactivate FBO_LF[v] set perspective v drawScene( ); // render desired 3Dscene (e.g., a teapot) end // factorize light field using NTF NTF( );end function NTF( ) for all iterations i // update the layers for alllayers l // draw current estimate of LF into rec buffersdrawLightFieldFromLayersRec( ); for all frames f // draw layers into LFtmp buffers, but leave out current layer drawLightFieldFromLayersTmp(l);// compute numerator for multiplicative NTF update activateFBO_LAYER_TMP[1] activate accumulation buffer for all light field viewsv set perspective i as projective texture matrix activateCG_SHADER_MULT2TEXTURES_AND_PROJECTIVE_TEX MAPTHEM( FBO_LF[v],FBO_LF_TMP[v] ) draw 2D quad end deactivate FBO_LAYER_TMP[1] // computedenominator for multiplicative NTF update activate FBO_LAYER_TMP[2]activate accumulation buffer forall light field views v set perspectivei as protective texture matrix activateCG_SHADER_MULT2TEXTURES_AND_PROJECTIVE_TEX MAPTHEM( FBO_LF_REC[v],FBO_LF_TMP[v] ) draw 2D quad end deactivate FBO_LAYER_TMP[2] // updatecurrent layer for current frame activate FBO_LAYERS[l][f] activateCG_SHADER_MULT2TEXTURES_DIVIDEBYOTHER (FBO_LAYERS[l][f],FBO_LAYER_TMP[1], FBO_LAYER_TMP[2]) draw 2D quad deactivateFBO_LAYERS[l][f] end end // update the backlight If bUseBacklight //draw current estimate of LF into rec buffersdrawLightFieldFromLayersRec( ); for all frames f // draw layers into LFtmp buffers, but leave out backlight drawLightFieldFromLayersTmp( ); forall views v set perspective i as protective texture matrix // computenumerator for multiplicative NTF update activate FBO_BL_TMP[1] activateCG_SHADER_MULT2TEXTURES_AND_PROJECTIVE_TEX MAPTHEM( FBO_LF[v],FBO_LF_TMP[v] ) draw 2D quad deactivate FBO_BL_TMP[1] // computedenominator for multiplicative NTF update activate FBO_BL_TMP[2]activate CG_SHADER_MULT2TEXTURES_AND_PROJECTIVE_TEX MAPTHEM(FBO_LF_REC[v], FBO_LF_TMP[v] ) draw 2D quad deactivate FBO_BL_TMP[2] //downsample the backlight TMP FBOs to backlight resolution by adding upthe values downsampleAndAdd(FBO_BL_TMP[1,2]); // update currentbacklight view for current frame activate FBO_BL[v][f] activateCG_SHADER_MULT2TEXTURES_DIVIDEBYOTHER ( FBO_BL[v][f], FBO_BL_TMP[1],FBO_BL_TMP[2] ) draw 2D quad deactivate FBO_BL[v][f] end end end end endAlgorithm NTF - Additional Helper Functions functiondrawLightFieldFromLayersRec( ) convertAllLayersAndBacklightToLOG( ); forall views v for all frames f activate FBO_TMP activate accumulationbuffer for all layers l draw layer l, textured with FBO_LAYERS[l][f] endIf bUseBacklight draw backlight, textured with FBO_BACKLIGHT[v][f] enddeactivate FBO_TMP activate FBO_LF_REC[v] activate accumulation bufferactivate CG_SHADER_DRAW_EXPONENTIAL_TEXTUE (FBO_TMP) deactivateFBO_LF_REC[v] end end convertAllLayersAndBacklightFromLOG( ); endfunction drawLightFieldFromLayersTmp(int leaveOutLayerX)convertAllLayersAndBacklightToLOG( ); for all views v for all frames factivate FBO_TMP activate accumulation buffer for all layers l IfleaveOutLayerX!=l draw layer l, textured with FBO_LAYERS[l][f] end endIf bUseBacklight && (leaveOutLayerX) draw backlight, textured withFBO_BACKLIGHT[v][f] end deactivate FBO_TMP activate FBO_LF_TMP[v]activate accumulation buffer activate CG_SHADER_DRAW_EXPONENTIAL_TEXTUE( FBO_TMP ) deactivate FBO_LF_TMP[v] end endconvertAllLayersAndBacklightFromLOG( ); end

What is claimed:
 1. A method comprising, in combination: (a) using abacklight to provide light to a display device, which display deviceincludes one or more spatially addressable, light attenuating layers;(b) using the layers to display a temporal sequence of frames; and (c)using one or more processors (i) to perform an optimization calculationto compute, for each respective frame in the sequence and eachrespective layer in the one or more layers, attenuation of the light atrespective pixels of the respective layer; and (ii) to output controlsignals to control the attenuation;  wherein (I) the optimizationcalculation includes at least one mathematical operation anN^(th)-order, rank-M tensor, where M is equal to the number of frames inthe sequence, and N is equal to the number of the layers, if thebacklight is uniform, and N is equal to the number of the layers plusone, if the backlight is directional, (II) the optimization calculationincludes applying a weighted nonnegative tensor factorization, and (III)either (A) the backlight is uniform and the number of the layers is atleast three or (B) the backlight is directional and the number of thelayers is at least one.
 2. The method of claim 1, wherein the backlightis directional.
 3. The method of claim 1, wherein the backlight isdirectional and the number of the layers is at least two.
 4. The methodof claim 1, wherein the backlight is uniform and the number of thelayers is at least three.
 5. The method of claim 1, wherein the tensoris sparse.
 6. The method of claim 1, wherein the display device isconfigured to produce an automultiscopic display.
 7. The method of claim6, wherein: (a) the automultiscopic display has one or more fields ofview; (b) the display device has a front display surface; (c) eachrespective field of view, out of the one or more fields of view, iscentered about a viewing axis; and (d) the method further comprises (i)dynamically varying the viewing axis of each field of view,respectively, including to orientations that are not normal to the frontdisplay surface, and (ii) tracking gaze or head position of a human userof the display device.
 8. Apparatus comprising, in combination: (a) adisplay device, which display device includes one or more spatiallyaddressable, light attenuating layers, which layers are configured todisplay a temporal sequence of frames; (b) a backlight, the backlightbeing configured to provide light to the display device; and (c) one ormore processors, the one or more processors being configured (i) toperform an optimization calculation to compute, for each respectiveframe in the sequence and each respective layer in the one or morelayers, attenuation of the light at respective pixels of the respectivelayer; and (ii) to output control signals to control the attenuation; wherein (I) the backlight is directional and the number of the layersis at least one, (II) the optimization calculation includes at least onemathematical operation on an rank-M tensor, M being equal to the numberof frames in the sequence, and (III) the optimization calculationincludes applying a weighted nonnegative tensor factorization.
 9. Theapparatus of claim 8, wherein the tensor has an order equal to N+1,where N is the number of the layers.
 10. The apparatus of claim 8,wherein: (a) the backlight comprises a lens and a spatially addressablelight modulating layer; (b) the lens has a focal length, and (c) thelight modulating layer is positioned at a distance from the lens, whichdistance is equal to the focal length.
 11. The apparatus of claim 9,wherein the number of the layers is equal to at least two.
 12. Theapparatus of claim 8, wherein the display device is configured toproduce an automultiscopic display.
 13. The apparatus of claim 8,wherein the tensor is sparse.
 14. The apparatus of claim 8, wherein theoptimization calculation optimizes based at least in part on perceptionerror metrics.
 15. The apparatus of claim 8, wherein the optimizationcalculation calculates a set of per pixel attenuations, which setminimizes error between a light field transmitted from the displaydevice and a light field that would be created by a target 3D scene. 16.The apparatus of claim 8, wherein at least one of the layers comprisesboth (i) optical elements configured to transmit light and (ii) opticalelements configured to emit light.
 17. Apparatus comprising, incombination: (a) a display device, which display device includes one ormore spatially addressable, light attenuating layers, the layers beingconfigured to display a temporal sequence of frames; (b) a backlight,the backlight being configured to provide light to the display device;and (c) one or more processors, the one or more processors beingconfigured (i) to perform an optimization calculation to compute, foreach respective frame in the sequence and each respective layer in theone or more layers, attenuation of light at respective pixels of therespective layer; and (ii) to output control signals to control theattenuation;  wherein (I) the backlight is uniform, (II) theoptimization calculation includes at least one mathematical operation onan N^(th)-order, rank-M tensor, M being equal to the number of frames inthe sequence and N being equal to the number of the layers, (III) theoptimization calculation includes applying a weighted nonnegative tensorfactorization, and (IV) the number of the layers is at least three. 18.The apparatus of claim 17, wherein: (a) the display device is configuredto produce an automultiscopic display, which display concurrently hasone or more fields of view; (b) the display device has a front displaysurface; (c) each respective field of view, out of the one or morefields of view, is centered about a viewing axis; and (d) the displaydevice is configured to dynamically vary the viewing axis of each fieldof view, respectively, including to orientations that are not normal tothe front display surface.
 19. The apparatus of claim 18, wherein theapparatus is configured to track gaze or head position of a human userof the display device.
 20. The apparatus of claim 17, wherein the tensoris sparse.